You can give it a try.
Await puppeteer.launch ({userDataDir: path.join (os.homedir (), '.aaa-data'),} add a unified cache folder, then all tab pages that jump to the same address can share a cache.

@mowatermelon
Thanks for your answer.

It wasn't bad as a temporary trick. But it has problems and of course the bug still exists.

By setting userDataDir, we can maintain the previous status and after solving the challenge manually, the next time Chrome opens, we can open the site without the challenge page.

But when the CloudFlare cookie session expires, everything goes back to the way it was before. If we use, for example, newPage(), the challenge will not be solved. Or if we reach the first tab with the help of pages()[0], everything is broken there and the site's challenge is not solved in any way.

Unless we repeat the trick again, i.e. temporarily remove the call to pages() and run Chrome once to pass the challenge. Then return that function to our code.

In general, it did not cure much pain!

same problem :(

I'm experiencing the same thing on other sites. What seems to be happening from my analysis, is the initial tab is "untouched" by puppeteer. So I can do whatever I want in the initial tab and I'm not detected. If I have puppeteer open it's own tab, and I manually take all actions in that tab, then I get detected as a bot. Alternatively, if I have puppeteer make the initial tab active then I take manual action, I'm detected as a bot.

So it seems any tab that gets "touched" by Puppeteer, gets detected somehow. When I look at Browser Fingerprinting, this seems to be the case. I'm not sure what they're using to detect Puppeteer even with stealth enabled.

creepjs (https://abrahamjuliot.github.io/creepjs/) is detecting puppeteer when you use any tab besides the start-up tab.

Take note that 205 lies are being detected, here's a sample:

I'm launching with the latest version of Google Chrome(Version 117.0.5938.92 (Official Build) (64-bit)) on Windows10 and am using the latest Puppeteer Node.js version alongside the stealth plugin with all evasions active.

Another creepjs image:

Any ideas about how Cloudflare is detecting Puppeteer?

I don't know the ins and outs of Puppeteer but I know they use the chrome dev tools protocol, the same as chrome dev tools.

However if I open https://nowsecure.nl/ with chrome dev tools open I can get through fine?

Unfortunately, this problem was very serious and acute.
And I had to use the service FlareSolverr. This is a proxy to bypass cloudflare and they use selenium.
I just send my first page to it and return only the cf_clearance cookie, set it to my puppeteer and continue...

Puppeteer can be detected by https://abrahamjuliot.github.io/creepjs/ if that helps. It's not just that it detects a bot, it detects PUPPETEER.

Unfortunately, this problem was very serious and acute. And I had to use the service FlareSolverr. This is a proxy to bypass cloudflare and they use selenium. I just send my first page to it and return only the cf_clearance cookie, set it to my puppeteer and continue...

For now this is a solution with no alternatives. Puppeteer or playwright + any anti-detection methods cannot solve problems with cloudflare.

I'll make a clarification.
If your ip or proxy is not on the list of suspicious ones, then there is an option to get challenge v1, which can be completed without a click and with the launch command await puppeteer.launch({ targetFilter: (target) => !!target.url() });
But if your ip is suspicious, then targetFilter: (target) => !!target.url() blocks work with the iframe and the possibility of manipulating the challenge checkbox disappears.

How is it that FlareSolverr works? Surely if Puppeteer is detected by Cloudflare then so would Selenium

How is it that FlareSolverr works? Surely if Puppeteer is detected by Cloudflare then so would Selenium

FlareSolverr uses undetected-chromedriver. UC uses completely different detection bypass methods, which are still difficult to replicate in puppeteer and puppeteer-extra.

Hello,
https://www.npmjs.com/package/cloudflare-scraper
With this bookshelf you can scrape and get cookies.

This doesn't make sense to me, I am using puppeteer to connect to chrome on android just connecting through the dev tools port, and even pages not created by puppeteer, as soon as they are "touched" cannot get past cloudflare

I have tried the constant page refresh problem when trying to switch to Cloudflare on the aforementioned sites. I have been repeatedly signing up on a different site for about 10 hours and they use cloudflare premium captcha. It passed all of them without any problem without needing to touch it. I don't see any problem in the package right now except fingerprint. Could you please try the package? I tried it on many sites like 000webhost openai etc. It was successful on all of them.

Unfortunately, this won't work for my use case as I launch chrome on android via ADB, forward the chrome dev tools port and then connect, so no launching is involved.

Hence why I am not sure this has fixed the issue, the only change I can see from the code in connecting is setting the user agent. If it works this is good news though

Unfortunately, this won't work for my use case as I launch chrome on android via ADB, forward the chrome dev tools port and then connect, so no launching is involved.

Hence why I am not sure this has fixed the issue, the only change I can see from the code in connecting is setting the user agent. If it works this is good news though

The important part to make it work is the cdp session which starts chrome on a port. We start a real browser on a port on the computer and connect with puppeteer's connect feature. The user agent is set to be the same as the created browser's agent. Here are some examples of it working:
https://www.youtube.com/watch?v=OgQOaVNTPa4
https://www.youtube.com/watch?v=vfzEHsoJpuw
https://www.youtube.com/watch?v=iTSVrtf8xXI

I will add linux support soon. If there is a site you want me to try, I can try it. In your case, after switching to cloudflare, the cookie can be taken and used with adb.

you guys can back to puppeteer version 5.5.0, all the problems will be solved, however, it quite old

Can I have your example package.json?

"puppeteer": "5.5.0", you should change to this version, I will share the detail problems later.

"puppeteer": "5.5.0", you should change to this version, I will share the detail problems later.

Why that version specifically?

Remove --single-process arg, cloudflare detects it.

I tried it, it didn't work. I used the current arguments on puppeteer version 9.1.1. and they worked then.
Thank you for your help. There are a lot of things to go through, I will start by downgrading my current version of puppeteer to 5.5.0 as previously recommended by #832 (comment).

For the experiment, try to upgrade it to the latest version (22) and test.

Cloudflare detects --single-process in iframe. Remove it

The IP you provided has a fraud score 100% and bot status true. Maybe they do not like the IP or the DC (pq hosting). But I think chances are low.

{
"success": true,
"message": "Success",
"fraud_score": 100,
"country_code": "KZ",
"region": "Almaty",
"city": "Almaty",
"ISP": "Stark Industries Solutions",
"ASN": 44477,
"organization": "Stark Industries Solutions",
"is_crawler": false,
"timezone": "Asia/Almaty",
"mobile": false,
"host": "kzfull.privateip.net",
"proxy": true,
"vpn": true,
"tor": false,
"active_vpn": false,
"active_tor": false,
"recent_abuse": true,
"bot_status": true,
"zip_code": "N/A",
"latitude": 43.25999832,
"longitude": 76.93000031,
"request_id": "OHwbkYSt9R"
}

I have a method that partially solves my problem. It involves running two browsers: one that I use locally to pass the Cloudflare check, and a second identical browser (with the same version) running remotely. This remote browser is proxied and uses my user data that I pass to it. This method works, but it is extremely inconvenient.

A browser initiated with Puppeteer always triggers an endless Cloudflare verification loop, whereas a manually launched browser does not have this issue. Clearly, Cloudflare detects the use of Puppeteer. Out of curiosity, I compared all properties of the window object in a headless browser and a regular browser. I did not notice any significant differences except for one that serves as an indicator of Puppeteer usage for websites. This is the document.createElement function. For some reason, Puppeteer uses a proxy version of this function instead of the standard one, which can be easily detected with the following code:

function isPuppeteer(){
return Object.getOwnPropertyNames(document).includes('createElement')
}

This needs to be fixed to make Puppeteer less detectable. Please help me reach out to the developers and comment on the discussion I opened: #puppeteer/puppeteer#12589, so this problem does not go unnoticed.

I get false in your example. Puppeteer 22.

You either did not use stealth.enabledEvasions.delete('iframe.contentWindow') or add additional command line args.

This issue has nothing to do with puppeteer. This lib (puppeteer-extra) sets proxy on createElement.
But only in headless mode.

Can you package all your files including package.json and upload here to reproduce?

I used these stealthPlugin methods, I also disabled all browser startup arguments.
The main condition is that the browser should not be connected to any user data (userDataDir). If you don't have a problem with Cloudflare, then maybe the site is picking up your pre-recorded user data.

The same browser works differently depending on whether I manually launched it or it was launched via puppeteer. This is most likely a puppeteer issue

If it is as you wrote, then I am wrong and need to remove puppeteer-extra because it leaves traces

Yes, it is true. puppeteer-extra is bad idea now. I do not use it for years.
But still possible to use. You probably do something wrong.
Need to see the full source code, with all files, like package.js

I can't forward the project. But the code in the project is very simple: I open a page with cloudflare protection and I can't go through it manually.
Disabling puppeteer-extra didn't help either

You have a bug somewhere.
I suspect in command line args.
Not possible to suggest without ability to reproduce.
Everything you sent works.

"puppeteer": "^9.1.1", "puppeteer-core": "^21.5.2", "puppeteer-extra": "^3.3.6", "puppeteer-extra-plugin-stealth": "^2.11.2",
I can't update the puppeteer, the new version has a bug that is not compatible with my project
puppeteer/puppeteer#12588

@vladtreny thanks for the help, you were right the --single-process argument was the problem. When I turned it off I didn't feel any difference at all, the looping continued. Now I tested again, this time I waited a bit longer and after a few iterations/cycles the captcha was passed. Now I can sleep well.

Remove --single-process arg, cloudflare detects it.

Please share what other arguments are undesirable